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ABSTRACT 

To address system problems and other management weaknesses, 
in 1998 Congress created the Office of Federal Student Aid ( FSA) within the 
Department of Education. This office adopted a new approach to systems 
integration using middleware (software that can allow an application to 
access data residing indifferent databases) and Extensible Markup Language 
(XML) , a flexible nonproprietary set of standards that is intended to make it 
easier to identify, integrate, and process information widely dispersed among 
systems and organizations. FSA first used this approach in the Common 
Origination and Disbursement (COD) Process for the Direct Loan, Pell Grant, 
and campus-based programs. This study was a follow-up review by the General 
Accounting Office (GAO) to assess FSA's progress in implementing this 
process. Data were from a variety of sources, including review of FSA 
documentation and reports from outside contractors. The GAO concludes that 
FSA has made progress in implementing the COD process, although 
implementation is behind schedule. The GAO has made recommendations to 
improve the process, and among these is the recommendation that the Secretary 
of Education direct the Chief Operating Officer, FSA, to establish a process 
to capture and disseminate lessons learned in schools. Appendixes contain a 
diagram of the COD process and comments from the FSA about the report. (SLD) 
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Highlights of GAO-03-241 , a report to the 
Secretary of Education 



Why GAO Did This Study 

To address system problems and 
other long-standing management 
weaknesses, in 1998, the Congress 
created a discrete unit within the 
Department of Education, the 
Office of Federal Student Aid 
(FSA). This office subsequently 
adopted a new approach to 
systems integration using 
middleware (a type of software 
that can allow an application to 
access data residing in different 
databases) and Extensible Markup 
Language (XML) — a flexible, 
nonproprietary set of standards 
that is intended to make it easier to 
identify, integrate, and process 
information widely dispersed 
among systems and organizations. 

FSA’s first use of this approach is 
the Common Origination and 
Disbursement (COD) process for 
the Direct Loan, Pell Grant, and 
campus-based programs. GAO 
initiated a follow-up review to 
assess FSA’s progress in 
implementing this process. 



What GAO Recommends 



Among GAO’s recommendations is 
that the Secretary of Education 
direct the Chief Operating Officer, 
FSA, to establish a process to 
capture and disseminate lessons 
learned to schools. 

In commenting on a draft of this 
report, FSA provided updated 
information and technical 
comments. GAO modified the 
report to address this new 
information, as appropriate. 
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FEDERAL STUDENT AID 

Progress in Integrating Pell Grant and 
Direct Loan Systems and Processes, but 
Critical Work Remains 



What GAO Found 

FSA has made progress in implementing the COD process. Specifically, it has 
implemented 

• a new information technology infrastructure that uses middleware to 
enable data exchange among disparate systems; 

• the initial version of the basic COD system, which replaces two existing 
systems and is being used by schools participating in the Pell Grant and 
Direct Loan programs; 

• middleware into existing systems to support the COD process; and 

• a common record based on XML that schools can use to submit student 
financial data for the Pell Grant and Direct Loan programs. 

However, the implementation of the COD process is behind schedule, and its 
ultimate success hinges on FSA’s completing critical work, including 
addressing serious postimplementation operational problems, and having 
thousands of postsecondary schools implement the common record. 

Further, there are important elements to managing any information 
technology investment that FSA has not yet completed: 

• Determining whether eocpected benefits are being achieved . As 
illustrated below, FSA has only some of the metrics, baseline data, and 
tracking processes necessary to determine whether it is achieving all 
expected benefits. 

• Tracking lessons learned . FSA has relied on an ad hoc approach for 
gathering and disseminating lessons learned related to schools’ 
implementation of the common record. To address this issue, FSA plans 
to include lessons learned as part of an update to its school testing guide. 
However, this does not replace the need for an ongoing mechanism to 
capture and disseminate lessons learned, without which schools may 
encounter problems that could have been avoided or mitigated. 



Status of the Office of Federal Student Aid Tracking of Actual Benefits of the COD System 


Expected benefit 


Metrics 

defined? 


Baseline data 
available? 


Benefits 

tracked? 


Reduced cost 


Yes 


Yes 


Yes" 


Increased customer satisfaction 


Partially^ 


Partially 


No 


Increased employee satisfaction 


N/A c 


N/A 


N/A 


Increased financial integrity 


Partially* 1 


Partially 


Partially 


Integration and modernization of legacy systems 


Yes 


Yes 


Yes 



"In providing comments on a draft of this report, FSA stated that it has begun tracking this expected 

benefit and provided supporting documentation, which we did not validate. 

b FSA stated that at this time, it did not have approval from the Department of Education to perform 

the surveys that it had planned to use to validate this benefit. However, FSA stated that it is trying to 

identify alternative metrics to measure customer satisfaction improvements due to COD. 

c Not applicable; although this was identified in the COD business case as an expected benefit, in 

commenting on a draft of this report, FSA stated that it is no longer using this as a COD benefit. 

d Not all metrics for this expected benefit have been defined. best copy availae 

Source: GAO analysis on the basis of FSA documentation. 

Q United States General Accounting Office 
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Accountability * Integrity * Reliability 



United States General Accounting Office 
Washington, DC 20548 



December 31, 2002 

The Honorable Roderick R. Paige 
The Secretary of Education 

Dear Mr. Secretary: 

Over the past decade, the Department of Education has spent millions of 
dollars to modernize and integrate 1 its disparate financial aid systems in an 
effort to improve the administration of its programs and provide more 
information and greater service to its customers — students, parents, 
schools, and lenders. However, as we reported last year, the department’s 
efforts had achieved limited success. 2 In January 1993, for example, the 
department awarded a 5-year, $39 million contract for the development 
and maintenance of the National Student Loan Data System, which was to 
provide information on systems across programmatic boundaries, yet it 
often lacks the most recent information from other financial aid systems. 

To address such systems problems and other long-standing management 
weaknesses, in 1998, as you know, the Congress created the federal 
government’s first performance-based organization, 3 currently called the 
Office of Federal Student Aid (FSA). 4 Under the performance-based 
organization concept, FSA is a discrete organizational unit within the 
Department of Education, led by its own Chief Operating Officer. FSA 
focuses on managing the operation of the student financial assistance 



'information integration is defined by the National Institute of Standards and Technology 
as the establishment of the appropriate computer hardware/software, methodology, and 
organizational environment to provide a unified and shared information management 
capability for a complex business enterprise. 

2 U.S. General Accounting Office, Student Financial Aid: Use of Middleware for Systems 
Integration Holds Promise, GAO-02-7 (Washington, D.C.: Nov. 30, 2001). 

3 The Higher Education Amendments of 1998, which amended the Higher Education Act of 
1965, states that the responsibilities of the performance -based organization include 
integrating the information systems supporting federal student financial assistance 
programs; implementing an open, common, and integrated system for delivery of student 
financial assistance under title IV; and developing and maintaining a student financial 
assistance system that contains complete, accurate, and timely data to ensure program 
integrity. 

4 Financial aid programs are administered by an office previously known as the Office of 
Student Financial Assistance, which was changed to the Office of Federal Student Aid on 
March 6, 2002. 
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programs, while the Department of Education focuses on policy-making 
functions. 

FSA subsequently adopted a systems integration approach that uses 
middleware 5 and Extensible Markup Language (XML) 6 technologies. In 
November 2001, we reported that in selecting middleware, FSA adopted a 
viable, industry-accepted mechanism for addressing its long-standing 
systems integration problems. 7 We also reported that FSA’s first use of 
middleware and XML as part of a process for delivering Direct Loan and 
Pell Grant aid to students was expected in early 2002. This process, called 
the Common Origination and Disbursement (COD) process, was expected 
to use a new system, also called COD, and be supported by a new 
information technology (IT) infrastructure and various existing systems. 

In the summer of 2002, we initiated a follow-up review to our November 
2001 report. Our objective was to assess FSA’s progress in implementing 
the COD process. In doing this work, we reviewed applicable FSA and 
Accenture (the prime contractor) documentation, including FSA’s 
modernization blueprint, the COD business case, the COD system’s 
requirements, the COD and Enterprise Application Integration 
implementation timelines, Accenture project status briefings, and test 
guides and results. In addition, we reviewed reports by an independent 
verification and validation contractor. We also interviewed applicable 
officials from FSA IT and program offices and Accenture. 

We performed our work at FSA headquarters in Washington, D.C., 
between August and November 2002, in accordance with generally 
accepted government auditing standards. 



^ddleware is a type of software that can allow an application to access data residing in 
different databases. In addition, middleware can enable dissimilar systems to communicate 
and work together as if they all resided on a single platform. 

®XML is a flexible, nonproprietary set of standards that offers the promise of making it 
significantly easier to identify, integrate, and process information that is widely dispersed 
among systems and organizations. 

7 GAO-02-7. 
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Results in Brief 



FSA has made progress in implementing the COD process. 8 Specifically, it 
has implemented (1) a new IT infrastructure that uses middleware to 
enable data exchange between disparate systems, (2) the initial version of 
the basic COD system, (3) middleware into existing systems to support the 
COD process, and (4) a common record based on XML that schools can 
use to submit student financial data for both the Pell Grant and Direct 
Loan programs. Nevertheless, the implementation of the COD process is 
behind schedule, and its ultimate success hinges on FSA’s completing 
critical work, including the implementation of the basic COD system 
requirements, addressing serious postimplementation operational 
problems, and having thousands of postsecondary schools implement the 
common record. 

Important elements of managing an IT investment are determining 
whether expected benefits are being achieved and tracking lessons 
learned. However, at this time, FSA is not fully tracking whether it is 
achieving certain expected benefits, such as increased customer 
satisfaction. In this instance, FSA COD officials stated that they did not 
have approval from the Department of Education to perform the survey 
that they had planned to use to validate this expected benefit, but they are 
trying to identify alternative metrics. Without such data, FSA lacks vital 
information on whether it is achieving all of its investment goals. 

Regarding lessons learned, FSA relies on an ad hoc approach to gathering 
and disseminating such information, which it believes is an adequate 
approach. However, such an ad hoc process may not ensure that all 
schools obtain these critical data. As a result, schools may encounter 
problems that could have been avoided or mitigated had they known of 
other schools’ experiences. 

We are making recommendations to improve the management of FSA’s 
COD program, including recommending that you direct FSA’s Chief 
Operating Officer to develop (1) metrics and baseline data that are related 
to increased customer satisfaction and increased financial integrity and 
(2) a tracking process to assess the extent to which the expected results 
are being achieved. 



8 Accenture was responsible for providing the project management, software, testing, and 
operation of the technology elements of the COD process. However, as used here, we 
ascribe the action to FSA because it made the decision to accept and deploy the 
contractor’s work. 
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In commenting on a draft of this report, FSA’s Chief Operating Officer 
provided updated information and technical comments, but did not 
comment on our recommendations. Among the comments provided, the 
Chief Operating Officer (1) stated that the draft report did not adequately 
portray the level of COD progress that had been made, (2) agreed that FSA 
was not tracking all expected benefits at this time and provided new 
information and supporting documentation on its recent efforts in this 
area, and (3) stated that FSA’s “informal” process for communicating 
lessons had worked well but planned to provide written lessons learned as 
part of a planned update of its school testing guide. We updated our report 
to reflect new FSA processes and information, as appropriate. However, 
we believe that we have accurately portrayed FSA’s progress, particularly 
in light of its need to address operational problems and facilitate the 
implementation of the common record at thousands of postsecondary 
schools. In addition, we continue to believe that FSA’s ad hoc processes 
for capturing and disseminating lessons learned do not provide assurance 
that it has captured and disseminated lessons learned related to schools’ 
implementation of the common record and could overlook important 
improvements that could be made. 



Background 



The Department of Education’s FSA manages and administers student 
financial assistance programs authorized under title IV of the Higher 
Education Act of 1965, as amended. These postsecondary programs 
include the William D. Ford Federal Direct Loan Program (often called the 
Direct Loan program), the Federal Family Education Loan Program (often 
called the Guaranteed Loan program), the Federal Pell Grant Program, and 
campus-based programs. 9 Annually, these programs together provide 
about $50 billion in student aid to approximately 8 million students and 
their families. 

During the past three decades, the Department of Education has created 
many disparate information systems to support these various student 
financial aid programs. In many cases, these systems — run on multiple 
operating platforms using different network protocols 10 and maintained 



9 Campus-based programs, which include the Federal Work-Study Program, the Federal 
Perkins Loan Program, and the Federal Supplemental Educational Opportunity Grant 
Program, are administered jointly by FSA and postsecondary educational institutions. 

10 For example, the FSA systems environment includes operating platforms, such as IBM 
OS/390 mainframe, Sun Solaris on Sparc, and Windows NT, and network protocols, such as 
the Transmission Control Protocol/Intemet Protocol and Systems Network Architecture. 
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and operated by a host of different contractors — were unable to easily 
exchange the timely, accurate, and useful data needed to ensure the 
proper management and oversight of the various student financial aid 
programs. For example, as we reported in 1997, neither the National 
Student Loan Data System nor other systems were designed for efficient 
access to reliable student financial aid information, since many systems 
were incompatible and lacked data standards and common identifiers. 11 In 
addition, because FSA used three separate systems to originate and/or 
disburse title IV funds, access to student and school data was fragmented 
and unreliable. As a result, FSA found it increasingly difficult to quickly 
access data to support day-to-day operational and management decisions, 
and schools could not easily access data to obtain a clear picture of the 
title IV student aid that had been disbursed. 

In September 1999, FSA issued its initial modernization blueprint, 12 which 
was subsequently updated in July 2000, to transform the title IV student 
financial aid systems using technology. COD is one of four school service 
business processes 13 in FSA’s blueprint and is intended to implement a 
simplified process for the operation of the Direct Loan and Pell Grant 
programs. According to FSA’s modernization blueprint, the common 
origination and disbursement process is composed of seven steps 
involving students, the Department of Education, and schools: (1) obtain 
applicant data, (2) determine eligibility, (3) determine award, (4) notify the 
Department of Education of the intent to disburse, (5) obtain funds from 
Education, (6) disburse funds to student, and (7) close out. A common 
process to support origination and disbursement is considered critical to 
FSA’s goal of achieving an enterprisewide solution that provides real-time 
data to students, schools, and financial partners via Web portals. 

To implement COD, FSA is using middleware and XML technologies. 
Specifically, middleware is being used to integrate FSA systems that 



"U.S. General Accounting Office, Student Financial Aid Information: Systems 
Architecture Needed to Improve Programs' Efficiency, GAO/AIMD-97-122 (Washington, 
D.C.: July 29, 1997). 

12 FSA’s modernization blueprint describes the agency’s business requirements, business 
and technical architecture, and sequencing plan. 

13 The other three processes are program eligibility, program support, and financial 
transactions. 
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support the COD process. 14 Traditionally, systems integration would 
require building separate point-to-point interfaces between every two 
applications. Although this approach can be effective, it creates several 
problems, such as (1) every connection between two applications requires 
custom programming; (2) a lot of connections have to be developed when 
there are multiple data sources; and (3) whenever the logic or data in one 
application changes, the accompanying interface often also needs to be 
altered. Middleware represents an alternative means to the traditional 
approach, and it can provide a quicker and more robust solution to 
systems integration. In essence, middleware separates the business 
application from the technical details of interapplication communications. 
Thus, middleware can simplify and reduce the number of interfaces for 
multiple systems because it can handle differences in data formats and 
record layouts. 

As part of the COD process, XML is being used to consolidate multiple 
legacy record formats previously used by schools to submit data on the 
Pell Grant and Direct Loan programs. 15 By using an XML-based common 
record, schools can transmit one file with all of the student’s data instead 
of submitting separate legacy records with redundant student and school 
information. 

Appendix I provides a high-level depiction of the systems and technologies 
supporting the COD process as of November 2002. As depicted, the COD 
system can translate or convert legacy records by using middleware. In 
addition, middleware has been built into several existing systems so that 
they can establish connectivity and exchange data with the COD system 
through a common IT infrastructure. This IT infrastructure, called the 
Enterprise Application Integration (EAI) bus, is also implemented using 
middleware to route data between systems in a correct format. In addition, 
as part of the COD process, some schools have begun submitting Pell 
Grant and Direct Loan data using the XML-based common record. 



14 For additional information on middleware and how it works, see GAO-02-7 and North 
Carolina Information Resource Management Commission, North Carolina Statewide 
Technical Architecture, which can be found at 
http://irm.state.nc.us/techarch/chaps/pdffiles/pdflist.htm. 

15 For additional information on XML and how it works, see U.S. General Accounting Office, 
Electronic Government: Challenges to Effective Adoption of the Extensible Markup 
Language, GAO-02-327 (Washington, D.C.: Apr. 5, 2002). 

10 
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FSA hired Accenture as its “modernization partner” to help carry out its 
modernization blueprint, including the implementation of the COD 
process. Accenture is the prime contractor 16 providing leadership of 
critical planning activities that are essential to the success of FSA’s 
modernization. Regarding the COD system part of FSA’s modernization, 
FSA also hired an independent verification and validation contractor to 
review the initial release of this system, which was completed earlier this 
year. 


Progress in 
Implementing the 
COD Process, but 
Critical Work 
Remains, and Benefits 
and Lessons Learned 
Are Not Being 
Tracked 


FSA has made progress in implementing the new COD process. In 
particular, it has begun implementing (1) its middleware solution in its IT 
infrastructure and various existing systems, (2) the COD system, and 
(3) an XML-based common record. However, FSA’s implementation of 
COD is behind schedule, and critical work remains to be completed. For 
example, the basic COD system was to be completed by mid-October 2002; 
however, only about three-quarters of the COD basic system requirements 
had been implemented as of October 23, 2002. In addition, FSA is not 
tracking whether it is achieving certain benefits because it is still in the 
process of defining applicable metrics to measure progress. Without such 
tracking processes, FSA lacks critical information about whether it is 
achieving expected benefits. Finally, FSA lacks assurance that it has 
captured and disseminated important lessons learned related to schools’ 
implementation of the common record because it believes that its current 
ad hoc process is adequate. Accordingly, the thousands of schools that 
have not yet implemented the common record may not benefit from the 
experience of those that have. 


FSA Has Begun 
Implementing COD 
Capabilities, but Critical 
Work Remains 


FSA has made progress in implementing COD. The following are 
significant elements of the COD process that have been implemented: 

Deployment of the EAI bus. As a prerequisite to implementing COD, in late 
October 2001, FSA deployed its middleware solution in an EAI “bus” — an 
IT infrastructure that uses middleware to access data from disparate 
systems, transform the data formats as necessary, and route the data to 
the appropriate requesting systems, thus enabling data exchange among 



O 

ERLC 


16 Accenture also uses subcontractors to perform some of the requirements related to this 
contract. 

1 1 

Page 7 GAO-03-241 Federal Student Aid Systems Integration 



disparate systems. The EAI bus provides the set of technical capabilities 
necessary to integrate FSA’s disparate systems. 

• Initial implementation of basic COD system ( release 1.x). 17 On April 29, 
2002, FSA went live with version 1.0 of the basic COD system. As of mid- 
November 2002, FSA had released an additional five sub-versions of the 
COD system (e.g., version 1.1). The COD system replaces the Direct Loan 
Origination System and the Recipient Financial Management System, and 
it currently processes files for all schools participating in the Pell Grant 
and Direct Loan programs. 18 According to FSA, in the first 6 months of its 
operation, the COD system processed just under 16 million transactions, 
representing Pell Grant and Direct Loan awards totaling almost $10 billion 
to over 5 million recipients. 

• Implementation of middleware in selected systems. As of mid-November 
2002, FSA had built middleware into seven systems so that these systems 
can interact with the COD system through the ELM bus. These systems 
include (1) the Central Processing System, which determines students’ 
eligibility and award levels, and (2) the National Student Loan Data 
System, which contains loan- and grant-level information and is used by 
schools to screen student aid applicants to identify ineligible borrowers. 

• Development and implementation of the common record. Using XML, 
FSA developed and began implementing a common record that schools 
can use to submit student financial aid data to the COD system. The 
common record, designed with assistance from members of the National 
Council of Higher Education Loan Programs and the Postsecondary 
Electronic Standards Council, consolidates multiple legacy file formats 
previously used by the Pell Grant and Direct Loan programs. 19 



iT “X” represents the sub-version number of the COD system. As of mid-November 2002, 

FSA planned to deploy a total of seven sub-versions of release 1.x. 

18 FSA plans to keep these two systems functioning until it (1) completes the reconciliation 
of prior years’ processing for direct loans, which is expected to occur by the end of this 
fiscal year, and (2) migrates the Recipient Financial Management System data to COD, 
which is expected by the summer of 2003. 

19 The common record also includes data blocks for campus-based programs to simplify 
school reporting. At this time, schools cannot report these data to FSA because the 
campus-based program part of the COD system is not yet available. According to a COD 
official, the campus-based data blocks in the common record are expected to be enabled in 
the summer of 2003. 

12 
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Although FSA has made progress in implementing the COD process, 
critical work remains to be completed. First, FSA is behind schedule in 
implementing the basic COD system. Although FSA had planned to 
complete the basic COD system by mid-October 2002, only about three- 
quarters 20 of the COD basic system requirements had been implemented as 
of October 23, 2002. 21 For example, as of early November 2002, one of the 
basic business functions that remains to be implemented is to enable FSA 
to make automated adjustments in batches to school current funding 
levels. FSA now estimates that most of the remaining functionality will be 
completed by the end of September 2003. According to FSA IT and 
program officials, the implementation of the basic COD functionality was 
delayed to allow adequate time for testing to ensure the quality of the 
system. 22 

Second, as of November 19, 2002, Accenture reported several operational 
problems that needed to be addressed. For example, in some cases, the 
COD system was incorrectly processing school batch data that contained 
multiple change records for an individual student. According to COD and 
contractor officials, the causes of operational problems included unclear 
requirements and software design defects. An independent verification 
and validation contractor also found problems with the requirements and 
design aspects of release 1.0. The COD Contracting Officer’s 
Representative characterized these operational problems as very serious 
and stated that they could impede operations and the delivery of future 
COD releases. This same official noted that FSA and Accenture are 
currently undertaking efforts to address these problems. For example, 

FSA has established production teams composed of agency and contractor 
staff to address problems in specific areas. In addition, FSA has 
established a continuous improvement process to more rigorously manage 
its relationship with Accenture. 

Third, fewer postsecondary schools than planned have implemented the 
common record. FSA had estimated that 50 schools (out of about 5,500) 
would implement the common record in fiscal year 2002. However, as of 



20 This percentage does not include functions initially planned to be included in the COD 
system but subsequently cancelled. 

21 In addition, on September 4, 2002, the COD Development Manager estimated that 
Accenture had met about 88 percent of the COD production acceptance criteria. 

“Once release 1.x is completed, FSA plans to implement other functionality, including 
campus-based program reporting. 
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November 26, 2002, only 22 schools 23 had implemented and tested the 
common record with FSA. FSA COD officials attributed the fewer-than- 
expected number of schools using the common record to schools and 
vendors not being ready to implement it. FSA expects that the number of 
schools using the common record will be considerably higher during the 
next award year (2003-2004) because, by April 2003, it plans to implement 
and test the common record with EDExpress, a software application FSA 
distributes free of charge to about 3,000 schools for use in submitting data 
In addition, FSA expects that all schools will be using the common record 
format by March 2004, in time for the 2004—2005 award year. 



FSA Is Not 
Completely Tracking 
Actual COD Benefits 
or Lessons Learned 
Related to Schools’ 
Implementation of the 
Common Record 



In its COD business case, FSA outlined five expected benefits: (1) reduced 
cost, (2) increased customer satisfaction, (3) increased employee 
satisfaction, (4) increased financial integrity, and (5) the integration and 
modernization of legacy systems. An important aspect of implementing an 
IT investment cited by the Office of Management and Budget 24 and our IT 
investment management guide 25 is evaluating the results of the investment 
by determining whether such expected benefits are being achieved. 
However, as illustrated in table 1, at this time FSA has only some of the 
data necessary to determine whether it is achieving all expected benefits. 
In particular, for the increased customer satisfaction and financial 
integrity benefits, FSA (1) has not fully defined the performance metrics to 
be used, (2) does not have all baseline data, and/or (3) is not fully tracking 
whether the benefits are being achieved. In these cases, FSA COD officials 
stated that they were in the process of developing relevant metrics, which 
would be tracked to measure the project’s performance against expected 
benefits. However, until FSA develops these data and begins tracking 
actual benefits and comparing them with expected benefits, it will lack 
vital data with which to demonstrate actual investment results. 



23 In addition, as of mid-October 2002, five vendors had implemented and tested the 
common record with FSA. Although FSA does not know how many schools are serviced by 
these vendors because the vendors consider this information proprietary, some of the 22 
schools that have implemented the common record use these vendors. 

24 Office of Management and Budget Circular A- 130, Management of Federal Information 
Resources (Nov. 30, 2000). 

25 U.S. General Accounting Office, Information Technology Investment Management: A 
Framework for Assessing and Improving Process Maturity, GAO/AIMD-10-1.23, Exposure 
Draft (Washington, D.C.: May 2000). 
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Table 1 : Status of FSA Tracking of Actual Benefits of the COD System 


Expected benefit 


Metrics defined? 


Baseline data available? 


Benefits tracked? 


Reduced cost 


Yes — The COD business case identifies 
decreased costs for systems operations, 
customer service, and the virtual data center. 


Yes 


Yes" 


Increased customer 
satisfaction 


Partially — The COD business case identifies 
the use of the American Customer Satisfaction 
Index scores to demonstrate clarity of 
instructions, ease of submitting data, and 
accuracy of records. However, FSA COD 
officials stated that at this time, they did not 
have approval from the Department of 
Education to perform this survey; therefore, 
they are trying to identify alternative metrics 
that could be used to measure customer 
satisfaction improvements due to COD. 


Partially — According to FSA, 
baseline data related to the 
American Customer 
Satisfaction Survey exist, 
but other planned metric(s) 
are not yet defined. 


No 


Increased employee 
satisfaction 


Not applicable — The COD business case 
identifies a metric for this expected benefit, 
but in commenting on our draft report, FSA 
stated that it has decided to discontinue the 
use of employee satisfaction as a key 
performance measure and thus will not be 
tracking this as an expected benefit of the 
COD system. 


Not applicable 


Not applicable 


Increased financial 
integrity 


Partially— -FSA COD officials stated that 
they have identified increased school 
compliance with its requirements that schools 
substantiate the amount of funds withdrawn 
within certain time frames as a metric. They 
also stated that they are currently developing 
other performance metrics for increased 
financial integrity but have not set a target 
date for completing them. 


Partially— Baseline data 
related to the identified 
metric exist, b but other 
planned metric(s) are not 
yet defined. 


Partially — Tracking the 
identified metric only. 


Integration and 
modernization 
of legacy systems 


Yes— The COD business case identifies a 
variety of quantitative and qualitative 
measures, including the retirement of two 
legacy applications and the simplification of 
testing and ongoing development. 


Yes 


Yes 



'In providing comments on a draft of this report, FSA’s Chief Operating Officer stated that after the 
completion of our review, FSA had begun tracking this expected benefit against the metrics defined in 
the business case. FSA also provided supporting documentation that previously was not available. 
We did not validate these new data. 

b ln providing comments on a draft of this report, FSA’s Chief Operating Officer stated that estimated 
baseline data for the identified metric are available, but FSA is working to develop more formal 
baseline data for this performance metric. 

Source: GAO analysis on the basis of FSA documentation. 



FSA IT officials also stated that they plan to have a contractor conduct a 
postimplementation review of the COD basic system in fiscal year 2003, 
which is expected to look at the achievement of expected benefits. While 
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this is an important initiative that could provide FSA with valuable 
information, it does not take the place of a continuing and systematic 
process of tracking actual benefits. 

According to our IT investment management guide, another critical 
activity is establishing a process for developing and capturing lessons 
learned in a written product or knowledge base and disseminating them to 
decision-makers . 26 Lessons-leamed mechanisms serve to communicate 
acquired knowledge more effectively and to ensure that beneficial 
information is factored into planning, work processes, and activities. 
Lessons learned can be based on positive experiences that save money or 
on negative experiences that result in undesirable outcomes. 

FSA has recognized the importance of generating lessons learned in 
certain areas. For example, it has implemented a process for developing 
lessons related to managing the relationship between the agency and its 
prime contractor. However, FSA lacks such a process for capturing or 
disseminating lessons related to school migration to the common record. 
FSA COD officials stated that lessons learned pertaining to school 
migration to the common record are addressed through periodic 
discussions during biweekly conference calls with schools undergoing 
testing with FSA and during portions of various FSA-sponsored 
conferences. FSA COD officials stated that they believed this process for 
capturing and disseminating lessons learned was adequate. 

However, by relying on such an ad hoc process, FSA lacks assurance that 
it has captured and disseminated all key lessons learned related to 
schools’ implementation of the common record and could overlook 
important improvements that could be made. In addition, schools that do 
not attend the conferences may not receive and benefit from the lessons 
identified in the initial phase of implementation. As a result, schools may 
encounter problems that could have been avoided or mitigated had they 
known of other schools’ experiences. This could hamper FSA’s ability to 
facilitate the transition of schools to the new common record and thus the 
agency’s ability to fully implement the new COD process and achieve the 
expected benefits. 

In commenting on a draft of this report, FSA stated that it plans to provide 
lessons learned as part of a planned update to its school testing guide. 



26 GAO/AIMD-10-1.23. 

16 
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While this is a positive step, it does not replace the need for mechanisms 
to continuously capture and disseminate acquired knowledge as schools 
implement the common record. 

Table 2 includes examples of lessons learned provided by FSA at our 
request that were drawn from schools’ initial implementation of the 
common record for the 2002-2003 award year. Such information would be 
important for the thousands of schools that have not yet implemented the 
common record so that they can avoid problems during the common 
record implementation and testing processes. 



Table 2: Examples of Lessons Learned Related to School Migration to the Common Record 


Lesson learned 


Effect 


Several schools and vendors did not have the technical 
staff to support the switch to XML. 


FSA had fewer schools participating in testing than initially expected. 


Many schools had only a technical contact or a business 
contact, but not both. 


Instructions FSA provided to schools may not be translated properly 
between the business and technical contacts. 


Schools did not sufficiently test their applications before 
testinq them with COD. 


Schools and vendors required more support from FSA than 
anticipated. 


Service-level expectations were not well-defined before 
the start of testing and were not made available to all 
parties involved. 


Schools had unrealistic expectations of the turnaround time required 
to process their records. 


Schools should use an XML tool to validate school XML- 
based applications before testing with the COD system. 


Schools that did not participate in such validations had more errors 
and a greater chance of their records being rejected upon submission 
to the COD system. 


Source: FSA. 



Conclusions 



the new COD process. However, critical actions, such as completing the 
basic functionality of the COD system and the implementation of the 
common record at thousands of affected schools, must still be undertaken. 
In addition, FSA has not yet fully established the metrics and processes to 
track actual benefits related to atl of its expected benefits or the lessons 
that have been generated by the few schools that have implemented the 
common record thus far. By not tracking actual benefits, FSA lacks 
information that is critical to determining whether it is meeting all of its 
goals. Further, not capturing and disseminating information to schools 
regarding lessons learned could make achieving these goals more difficult. 



Recommendations 



To determine the extent to which the new COD process is achieving 
expected results related to customer satisfaction and financial integrity, 
we recommend that you direct FSA’s Chief Operating Officer to 
expeditiously develop metrics and baseline data to measure these benefits 
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and develop a tracking process to assess the extent to which the expected 
results are being achieved. 

To ensure that the schools that have not yet implemented the common 
record benefit from the experiences of those that have, we recommend 
that you direct FSA’s Chief Operating Officer to establish a process for 
capturing lessons learned in a written product or knowledge base and for 
disseminating them to these schools. 



Agency Comments 
and Our Evaluation 



In providing written comments on our draft report, FSA’s Chief Operating 
Officer provided technical comments and updated information, but did not 
comment on our recommendations. Specifically, 

The Chief Operating Officer did not believe the report adequately 
portrayed the level of COD progress that had been made. In particular, she 
took issue with our using the completion of 75 percent of COD 
requirements as an indication of progress. Although the Chief Operating 
Officer did not disagree with the accuracy of this figure, she stated that 
FSA’s informal analysis indicated that between 85 to 90 percent of COD 
functions had been implemented, which she believed was a better gauge of 
progress. We believe that we have accurately portrayed FSA’s progress in 
implementing the COD process. First, since FSA’s analysis was “informal,” 
and FSA’s supporting documentation had limited detail that we could not 
validate, we do not agree that this should be the primary basis for an 
analysis of COD’s progress. Second, we included both the percentage of 
COD’s requirements that had been implemented and FSA’s estimate in our 
report. Nevertheless, we modified our report to include additional data 
provided by FSA regarding the number of transactions processed by the 
COD system to further indicate progress. 

The Chief Operating Officer agreed that the tracking of all of the expected 
benefits is not in place at this time, but stated that work is under way in 
this area. FSA also provided updated information and supporting 
documentation related to the tracking of some of the expected benefits. 

We made changes to the report reflecting this new information, as 
appropriate. 

The Chief Operating Officer agreed that it is important and beneficial to 
communicate lessons learned, but stated that FSA’s informal method for 
communicating lessons related to school migration to the common record 
worked well in the first year of COD implementation. FSA also noted that 
it plans to include lessons learned in a planned update to its school testing 
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guide. We modified the report to reflect this initiative, but we do not agree 
that FSA’s informal method or its plan to include lessons learned in its 
testing guide is adequate because these approaches do not provide a 
continuous process for actively capturing and disseminating lessons 
learned. As a result, some important lessons may be overlooked, and all 
schools may not be aware of potential problems associated with 
implementing the common record. 

FSA’s written comments, along with our responses, are reproduced in 
appendix II. 



We are sending copies of this report to the appropriate congressional 
committees, the Secretary of Education, the Chief Operating Officer of 
Education’s Office of Federal Student Aid, and the Director of the Office of 
Management and Budget. This report will also be available at no charge on 
the GAO Web site at http://www.gao.gov. 

If you have any questions on matters discussed in this report, please 
contact me at (202) 512-9286 or Linda J. Lambert, Assistant Director, at 
(202) 512-9556. We can also be reached by E-mail at pownerd@gao.gov and 
lambertl@gao.gov, respectively. Other individuals making key 
contributions to this report included Jason B. Bakelar and Anh Q. Le. 

Sincerely yours, 




David A. Powner 

Director (Acting), Information Technology 
Management Issues 
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Appendix I: Systems and Technologies 
Supporting the Common Origination and 
Disbursement Process, as of November 2002 




Source: GAO anafysis of FSA documents. 



‘This is a temporary interface. DLOS is targeted to be retired in fiscal year 2003. 
b MQSeries is IBM’s proprietary message format. 
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Appendix I: Systems and Technologies 
Supporting the Common Origination and 
Disbursement Process, as of November 2002 



'This is an information technology infrastructure that enables data exchange among disparate 
systems. 
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Appendix II: Comments from the Department 
of Education’s Office of Federal Student Aid 



Note: GAO comments 
supplementing those in 
the report text appear at 
the end of this appendix. 



See comment 1 . 



See comment 2. 



See comment 3. 




UNITED STATES DEPARTMENT OF EDUCATION 

Federal Student Aid 
Chief Operating Officer 



December 1 8, 2002 



Mr. David A. Powner 
Director (Acting), 

Information Technology 
Management Issues 

United States General Accounting Office 
Washington, DC 20548 

Dear Mr. Powner: 

I am writing in response to your request for comment on the draft GAO report to Secretary Paige 
on the implementation of the Common Origination and Disbursement (COD) system by Federal 
Student Aid. The draft report is entitled, “Federal Student Aid: System Integration Progressing, 
But Critical Work Remains.” Our understanding was that the purpose of this audit was refocused 
to specifically examine FSA’s progress in implementing the Common Origination and 
Disbursement System (COD), and did not include an examination of FSA’s system integration 
plan nor progress against that plan. Thus, we suggest that GAO change the title to make it clear 
that the audit was limited to an examination of the implementation of COD. In addition, we ask 
you to consider the information provided below as you finalize this report. 

Highlights, Results in Brief and Background 



In the “Highlights” section, GAO makes reference to “serious post-implementation operational 
problems”. However, no such problems are identified in the report. Further, in the “Results in 
Brief’ section on page 2 of the report, while GAO states that FSA has made progress in 
implementing the COD process, and describes four very significant accomplishments, the 
information that follows is out of context, and suggests that progress made has been minimal. 
Progress of significance that should be reflected in the report is that in the first six months of 
COD operations, the system has processed just under 16 million transactions. Those transactions 
report Pell Grant and Direct Loan awards totaling almost $10 billion to over 5 million recipients. 
While we certainly acknowledge that the system has had problems, the fact is that we have 
processed millions of records for billions of dollars, and have given the right money to the right 
people. 

In addition, the report on page 3 states that “...the COD process is behind schedule, and its 
ultimate success hinges on FSA’s completing critical work, including having thousands of post- 
secondary schools implement the common record.” This statement does not take into 
consideration that all post-secondary educational institutions participating in the Pell Grant 
and/or Direct Loan Programs (approximately 5,500) are current users of the COD system, 



Our mission i.s to ensure equal access to education and to promote educafionol excellence throughout the nation. 
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of Education’s Office of Federal Student Aid 



See comment 4. 



See comment 5. 



whether common record users or not. We believe that the GAO report should note that 
middleware functionality enables FSA to process all Pell Grant and Direct Loan record formats 
so that our schools can implement the XML formatted common record as their schedule permits. 
The common record, while new and innovative is only one of the important parts of the COD 
and it has always been our published intention that schools migrate to using the common record 
over a three-year period. Since the inception of the Commonline file format, which was first 
introduced many years ago, it has been standard student aid industry practice to have a planned 
phase-in of new file formats and processes. This is widely viewed as risk mitigation, and in fact, 
the industry supports that mitigation by supporting several versions of file formats while its 
customers adapt to the new process. GAO seems to be implying that phasing in the 
implementation of the common record was actually an INCREASE in risk, while FSA feels it 
was prudent as a way to DECREASE risk. 

The report on page 3 also discusses in brief the findings of the review that we shall address in the 
appropriate more detailed section of the draft report. This discussion, as well as the Highlights 
section, should be revised to reflect changes as a result of the comments that follow. 

Progress In Implementing COD Process, But Critical Work Remains and Benefits and 
Lessons Learned Not Being Tracked 

On page 6 and again on page 8 of the report, GAO states that as of October 22 only about three 
quarters of the COD basic system requirements had been implemented. This is based on an 
arithmetic calculation using the number of requirements implemented versus the total number of 
requirements. We believe it is important to note the relative functional value of the implemented 
and non-implemented requirements, because in fact, if only 75 percent of the critical COD 
functionality were in place it is doubtful that so many Pell Grants and Direct Loans could have 
been processed through the current date. 

As previously discussed with GAO staff, FSA COD IT development managers performed an 
informal analysis of the percentage of COD functionality that was implemented by the end of 
September 2002, as well as provided an analysis of the requirements in the initial 1 .0 COD 
software release against the contract’s 1.0 acceptance criteria. In both cases, the development 
managers provided an estimate that between 85-90 percent of the functionality was implemented 
and that the critical functionality necessary to process records had been implemented. We have 
replicated virtually all of the existing functionality of the two previous systems into an integrated 
common system, as well as provided a number of enhancements that please our customers and 
improve financial integrity. It seems to FSA that our goal of migrating existing systems into a 
common system has been met, although we recognize that there is support functionality that still 
needs to be completed, 

FSA Not Completely Tracking Actual COP Benefits or Lessons Learned Related to 
Schools’ Implementation of the Common Record 

On pages 3, 6, and in detail beginning on page 9 of the report, the GAO states that FSA is not 
tracking whether it is achieving certain expected benefits (as discussed in the FSA COD system 
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See comment 6. 



business case). We agree that all of the tracking of the expected benefits is not in place at this 
time but work on all is underway. The following is an update to each of the items listed in Table 
1 on page 11. 

Reduced Cost. The Contracting Officer’s Representative for the COD system has been tracking 
monthly COD costs for the 6 months of COD operations and has compared those costs to the 
baseline legacy system costs. However, as noted in the report, those costs were not broken down 
into the component business case costs of Operations, Customer Service, and virtual data center 
costs at the time of the GAO review team data collection. As part of our response, we are 
providing an excel spreadsheet (Attachment 1) containing such a comparison for the five months 
of COD operations in FY 02 with the understanding that this is an on-going tracking of this 
expected benefit. 

Increased Customer Satisfaction. Although FSA has for several years conducted a valid and 
reliable survey of its customers, the Department intends to modify the approach to, among other 
things, eliminate duplicate surveys of the Department’s customers. During this transition, no 
survey was conducted in FY2002, so we have no data for this indicator. However, we certainly 
intend to measure customer satisfaction, and as you noted, we are in the process of identifying 
alternative metrics that could be used to measure customer satisfaction improvements due to 
COD and we agree that it may be necessary to alter the COD contract to do so. 

Increase Employee Satisfaction. FSA has decided to discontinue employee satisfaction as a 
key performance measure for the organization and therefore, will not be using this activity as an 
expected benefit of the COD system. 

Increased Financial Integrity. FSA is tracking the performance of participating institutions 
against the key financial requirement of reporting actual disbursements of Pell Grants and Direct 
Loans within 30 days of drawing the funds down. GAO was provided with the procedures being 
employed by the COD Customer Service Center to monitor and act on the resulting performance 
data related to this key financial integrity measure. We are attaching a copy of the most recent 
weekly monitoring report (Attachment 2). This report demonstrates for the week of 12/02/02- 
12/06/02, a total of 421 of 5,500 (approximately 7.6%) participating schools are outside the 30- 
day requirement. We believe that this result is very noteworthy given that we have deliberately 
not aggressively pursued the schools compliance with this requirement this early into the 
implementation of COD. This 7.6% is compared to an estimate of 15-20% of schools being non- 
compliant with that requirement for any given period under the legacy systems. FSA is working 
to develop a more formal baseline set of numbers for this performance measure. 

Lessons Learned Related to School Testing. On page 12 in the report, GAO took issue with 
the manner in which lessons learned during school testing were being communicated to the 22 
full participant schools. FSA clearly understands the importance and benefits that can be derived 
by communication of lessons learned. FSA employed a more informal method of 
communicating lessons learned because of the limited number of full participant schools in this 
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See comment 7. 



See comment 8. 



inaugural year of COD implementation. This process worked well for this year, but FSA is 
providing a more formal method of disseminating this type of information to full COD 
participant schools that will test in the upcoming 03-04 COD year. GAO was provided with a set 
of slides used at the most recent FSA sponsored Electronic Access Conference held this month. 
One of the slides was labeled School Testing Lessons Learned (Attachment 3). In addition, the 
03-04 School COD Testing Guide under development will have a section on lessons learned 
from the schools and financial aid software vendors that tested with COD in the 02-03 COD year. 
We expect that document to be available in January 2003. 

Thank you for this opportunity to comment on the draft report on the implementation of COD. 




Theresa S. Shaw 
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of Education’s Office of Federal Student Aid 



GAO Comments 



ERJC 



The following are GAO’s comments on the Department of Education’s 
Office of Federal Student Aid’s letter dated December 18, 2002. 



1. We revised the draft report title to clarify that our follow-up review 
was focused on assessing FSA’s progress in implementing the COD 
process. 

2. Information related to operational problems was contained in the draft 
report. We asked the COD Contracting Officer’s Representative to 
characterize these problems, and he stated that they were very serious. 
In addition, we confirmed the seriousness of these problems at the 
conclusion of our review with FSA IT and program officials. 

3. We modified this report to include data on the number of transactions 
processed. We also modified our report to clarify that all schools 
participating in the Pell Grant and/or Direct Loan programs currently 
use the COD system. 

4. We do not agree that the report implies that FSA’s use of a phased-in 
approach in implementing the common record increases risks. Instead, 
this report notes that the implementation is not yet complete. 

5. We believe that we have accurately portrayed FSA’s progress in 
implementing the COD process. First, since FSA’s analysis was 
“informal,” and FSA’s supporting documentation had limited detail that 
we could not validate, we do not agree that the COD Development 
Manager’s functionality estimate should be the primary basis for an 
analysis of COD’s progress. Second, we included both the percentage 
of COD’s requirements that had been implemented and FSA’s estimate 
in our report. 

6. We modified our report to reflect this updated information as 
appropriate. 

7. We do not agree that FSA’s informal process for capturing and 
disseminating lessons learned was adequate because (1) it may lead to 
important lessons being overlooked and (2) all schools may not be 
aware of potential problems associated with implementing the 
common record. 

8. We modified this report to reflect that FSA plans to include lessons 
learned in a planned update to its school testing guide. While this is a 



Page 22 



"26 



GAO-03-241 Federal Student Aid Systems Integration 



Appendix II: Comments from the Department 
of Education’s Office of Federal Student Aid 



positive step, it does not replace the need for mechanisms to 
continuously capture and disseminate acquired knowledge as schools 
implement the common record. 
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GAO’s Mission 


The General Accounting Office, the investigative arm of Congress, exists to 
support Congress in meeting its constitutional responsibilities and to help 
improve the performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates federal 
programs and policies; and provides analyses, recommendations, and other 
assistance to help Congress make informed oversight, policy, and funding 
decisions. GAO’s commitment to good government is reflected in its core values 
of accountability, integrity, and reliability. 


Obtaining Copies of 
GAO Reports and 
Testimony 


The fastest and easiest way to obtain copies of GAO documents at no cost is 
through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- 
text files of current reports and testimony and an expanding archive of older 
products. The Web site features a search engine to help you locate documents 
using key words and phrases. You can print these documents in their entirety, 
including charts and other graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site 
daily. The list contains links to the full-text document files. To have GAO e-mail 
this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily 
E-mail alert for newly released products” under the GAO Reports heading. 


Order by Mail or Phone 


The first copy of each printed report is free. Additional copies are $2 each. A 
check or money order should be made out to the Superintendent of Documents. 
GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a 
single address are discounted 25 percent. Orders should be sent to: 

U.S. General Accounting Office 
441 G Street NW, Room LM 
Washington, D.C. 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 


To Report Fraud, 
Waste, and Abuse in 
Federal Programs 


Contact: 

Web site: www.gao.gov/ffaudnet/fraudnet.htm 
E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470 


Public Affairs 


Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 
U.S. General Accounting Office, 441 G Street NW, Room 7149 
Washington, D.C. 20548 
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